SSAST: Self-Supervised Audio Spectrogram Transformer

Recently, neural networks based purely on self-attention, such as the Vision Transformer (ViT), have been shown to outperform deep learning models constructed with convolutional (CNNs) various vision tasks, thus extending success of Transformers, which were originally developed for language processing, domain. A recent study showed that a similar methodology can also be applied audio Specifically, Audio Spectrogram (AST) achieves state-of-the-art results classification benchmarks. However, pure tend require more training data compared CNNs, and AST relies supervised pretraining requires large amount labeled complex pipeline, limiting practical usage AST. This paper focuses speech classification, aims reduce need amounts by leveraging self-supervised using unlabeled data. we propose pretrain model joint discriminative generative masked spectrogram patch modeling (MSPM) from AudioSet Librispeech. We evaluate our pretrained both tasks including event keyword spotting, emotion recognition, speaker identification. The proposed framework significantly boosts performance all an average improvement 60.9%, leading or even better than To best knowledge, it is first patch-based in domain,